System for maintaining coherency during offline changes to storage media

ABSTRACT

A proxy device monitors memory access operations between clients and a storage media. The proxy device stores at least some of the data from the storage media in a faster tiering media and provides the data from the faster tiering media to the clients for certain associated memory access operations. The proxy is also configured to monitor Small Computer System Interface (SCSI) communications between the clients and the storage media and invalidate at least some data in the tiering media when particular SCSI messages in the SCSI communications indicate the tiering media contains data that is out of sync with the data in the storage media.

RELATED APPLICATIONS

This application is continuation of U.S. patent application Ser. No.12/794,057 filed on Jun. 4, 2010 which is a continuation in part of U.S.patent application Ser. No. 12/619,609 filed Nov. 16, 2009 which claimspriority to U.S. provisional patent application Ser. No. 61/115,426,filed Nov. 17,2008, and which are both herein incorporated by referencein their entirety. U.S. patent application Ser. No. 12/794,057, to whichpriority is claimed herein, is also a continuation in part of U.S.patent application Ser. No. 12/568,612 filed on Sep. 28, 2009, now U.S.Pat. No. 8,160,070, which claims priority to U.S. ProvisionalApplication Ser. No. 61/101,645 filed Sep. 30, 2008, which are alsoincorporated by reference in their entirety.

BACKGROUND

Fibre Channel (FC) provides practical and expandable means oftransferring data between workstations, mainframes, supercomputers,desktop computers, and storage devices at fast data rates. Fibre Channel(FC) is especially suited for connecting computer servers to sharedstorage devices and for interconnecting storage controllers and drives.

A proxy device may be connected by a FC network between a clientcomputer and a storage device. The proxy device may contain a tieringmedia that needs to maintain an identical state as the storage device,so that consistent and correct data can be provided to the clientcomputer. However, the proxy device may not have access to alloperations performed on the storage device. These “off line” operationsmay leave different versions of data in the tiering media and in thedata storage device. When the client computer goes back “on line” andtries to access the storage device through the proxy device, the proxydevice may provide incorrect data from the tiering media unless madeaware of the offline activity.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a storage access system coupled between client devices andstorage media;

FIGS. 2 and 3 show how data snapshots are performed for data containedin the storage media of FIG. 1;

FIG. 4 is a flow diagram showing Small Computer System Interface (SCSI)operations performed for the snapshot operations of FIGS. 2 and 3;

FIG. 5 shows how the storage access system in FIG. 1 uses SCSIoperations to identity snapshot operations;

FIG. 6 is a flow diagram showing how the storage access system in FIG. 1invalidates data in a tiering media responsive to a SCSI bus rescan; and

FIG. 7 is a flow diagram showing how the storage access system in FIG. 1invalidates data in a tiering media responsive to a SCSI bus rescan anda SCSI device inquiry.

DESCRIPTION

Several preferred examples of the present application will now bedescribed with reference to the accompanying drawings. Various otherexamples are also possible and practical. This application may beexemplified in many different forms and should not be construed as beinglimited to the examples set forth herein.

FIG. 1 shows a storage access system 100 connected between clientdevices 106 and a storage media 114. The client devices 106 can beservers, personal computers, terminals, portable digital devices,routers, switches, or any other wired or wireless computing device thatneeds to access data on storage media 114. The client devices 106conduct different storage operations 102 with the storage media 114though the storage access system 100. The storage operations 102 mayinclude write operations 102A and read operations 102B. The storagemedia 114 may contain multiple media devices 120, such as multiplestorage disks that are referred to generally as a disk array.

In one embodiment, the storage access system 100 and the storage media114 are stand-alone appliances, devices, or blades. In one embodiment,the client devices 106, storage access system 100, and storage media 114might be coupled to each other via wired or wireless connections 112capable of transporting the storage operations 102 and any associateddata between client devices 106 and storage media 114.

One example of a connection 112 is a Fibre Channel network that uses theSmall Computer System Interface (SCSI) protocol for storage operations.Client devices 106, storage access system 100, and storage media 114 mayuse fibre channel interface cards or Host Bus Adapters (HBA) (notshown). The fibre channel HBAs allow the client devices 106 and storagemedia 114 to communicate over the fibre channel medium 112 using theSCSI protocol. Most FC networks utilize SCSI as the underlying storageprotocol, and any non-SCSI disk, such as a Serial ATA (SATA) disk,within storage media 114 will typically be virtualized as a SCSI entity.

In another embodiment, the client devices 106 may access one or more ofthe media devices 120 in storage media 114 over an internal or externaldata bus. The storage media 114 in this embodiment could be located inpersonal computers or servers, or could also be a stand-alone devicecoupled to the client computer/server 106 via a fiber channel SCSI bus,Universal Serial Bus (USB), or packet switched network connections 112.

The storage access system 100 contains one or more processors orprocessing elements 105 that operate as a proxy for the storageoperations 102 between the client devices 106 and storage media 114.Tiering media 110 in storage access system 100 includes differentcombinations of Flash memory and Dynamic Random Access Memory (DRAM)that typically provides faster access speeds than say disks that may beused in storage media 114.

The storage access system 100 receives the read and write operations 102from the client devices 106 that are directed to the storage media 114.In one embodiment, the media devices 120 contain multiple storage blocksthat have associated block addresses. To improve throughput and/or toreduce latency to the data in the storage media 114, some of the blocksof data from the storage media 114 are temporarily copied into thetiering media 110. The storage access system 100 then uses the data inthe faster tiering media 110 to service certain storage accessoperations 102 from the client devices 106.

In order to maintain data coherency, storage access system 100 monitorsall of the storage operations 102 performed in storage media 114 andmaintains the same version of data in the tiering media 110 and storagemedia 114. Proxy 105 is responsible for maintaining this data coherencybetween the tiering media 110 and the storage media 114 and must see allwrite operations to storage media 114.

FIGS. 2 and 3 show how snapshot operations might be performed in thestorage media 114. A snapshot operation is used for capturing data instorage media 114 at a particular instance in time. A client application116 operating on client device 106 may need to conduct a backupoperation for the data currently stored in storage media 114 or may needto process the data in storage media 114 as of a particular time. Forexample, a client database application 116 may need to generate reportsfor stock market transactions from the previous day. Stock transactionsare used as an example below, but of course any type of softwareapplication and data may be used.

The client device 106 uses a storage controller 130 to capture a stablestate or “snapshot” for the stock transactions from the previous tradingday. The storage controller 130 copies a particular set of snapshot datafrom storage media 114 into other media devices 119 or to a differentlocation in storage media 114. The storage media containing the snapshotdata is referred to generally as snapshot storage media 118 and is shownseparately from storage media 114 in FIGS. 3 and 4 for illustrationpurposes. However, the snapshot storage media 118 could be a particulardirectory or particular media devices 119 within the same storage media114.

After the snapshot operation, real-time read and write data can continueto be accessed in storage media 114 while the stock transactions fromthe previous day are isolated as read only data in snapshot storagemedia 118. The client database application 116 is then free to generatereports for the stock transactions from the previous day from snapshotstorage media 118. The advantage of this method is that snapshot storagemedia 118 will not be constantly updated with new transaction data andthus have superior performance from the perspective of client databaseapplication 116. Reports run against storage media 114 would generatethe same result, but will include content with real-time updates andthus be slower.

FIG. 3 shows an alternative embodiment where the storage controller 130generates a logical snapshot using pointers 122 in the snapshot storagemedia 118. Instead of copying all of the related data from storage media114 into snapshot storage media 118, the storage controller 130generates pointers 122 that point to data in the storage media 114 thathas not changed since the last snapshot operation. However, any data 124that has changed since the last snapshot operation is copied from thestorage media 114 into the snapshot storage media 118. Again, this couldcomprise the storage controller 130 copying the stock transactions fromthe previous day into a particular read only directory in the storagemedia 114 reserved for the snapshot pointers 122 and changed snapshotdata 124. To minimize the time required to perform the snapshotoperation, the snapshot method chosen often reflects the percentage ofsnapshot data that is dynamically changed within the real-time storage.If little change is expected, such as the previous day's stocktransaction data that is not expected to be modified during the currentday, a pointer system is usually more efficient.

The storage controller 130 needs to ensure that the data in snapshotstorage media 118 is accurate with respect to a particular point intime. Data operations should not be in transit when the snapshotoperations are performed. For example, the client application 116 shouldnot be performing account balance updates for the stock transactions forthe previous day while the storage controller 130 is generating thesnapshot data in media 118. Otherwise, the account balance updates maybe inconsistent with the stock transactions in snapshot media 18.Specifically, the snapshot operation may be performed within storagecontroller 130 and not be visible to Storage Access System 100 as nowrite operations are performed as the snapshot is created.

FIG. 4 shows how data is isolated during a snapshot operation. In block300 the client application 116 is shut down to temporarily stop any reador write operations 120 to storage media 114. The client device 106 inblock 302 unmounts the media devices 120 in the storage media 114. Forexample, the client device 106 may send unmount commands to itsoperating system. The unmount commands also clear any data that might becached in the client device 106, such as within the operating systemblock cache. The storage controller 130 in block 304 then logicallyremoves all media devices 120 from the SCSI network 112 using the methodsupported by the client device operating system. By clearing its caches,client device 106 assures data integrity when the devices are eventuallyrestored. The client application may have its own caches which arecleared upon shut down.

In block 306 the storage controller 130 is then free to perform thesnapshot operations described above in FIGS. 2 and 3 without the clientdevices 106 or media device 120 changing any data. After the snapshotdata is successfully copied into snapshot media 118, the storagecontroller 130 in block 308 adds the media devices 120 back to the SCSIbus 112 by requesting the client operating system to rescan the SCSI busand add available devices. In most cases, these new devices will havethe same identities as those unmounted in 302. The application thusrequires no change or reconfiguration, a key advantage of the snapshotprocess.

The client device 106 in block 310 remounts the media devices 120 forexample by sending mount requests to the client operating system. Theclient application 116 is then restarted on the client device 106 inblock 312. The client application 116 can then go back to performingreal-time write and read operations 102 with the storage media 114. Theclient database application 106 can also start generating the stocktransaction reports for the previous day from the data in snapshotstorage media 118.

One of the problems with these snapshot operations or any other offlineoperations, is that data is changed or updated by the storage controller130 offline from the read and write operations that normally passthrough storage access system 100. Because the storage access system 100cannot monitor these snapshot operations, the proxy device 105 cannotkeep the data in tiering media 110 coherent with the data in storagemedia 114. Other than the rescan operation, client requests to itsoperating system to mount and unmount devices are not visible on thestorage interface.

For example, the tiering media 110 may currently contain some of thesnapshot data for stock transactions that happened two days ago.However, after the snapshot operations in FIGS. 2 and 3, the snapshotstorage media 118 contains the stock transactions from one day ago,while the tiering media 110 still contains the stock transactions fromtwo days ago. If the data in tiering media 110 is not invalidated orcleared, the storage access system 100 may provide some of the two dayold data to the client application 116 instead of the one day old datain snapshot storage media 118. Because the snapshot operations wereconducted offline by the storage controller 130, the storage accesssystem 100 has no way of knowing if or when to clear tiering media 110.

Selectively Clearing Tiering Media

Table 1 below shows two control operations conducted using the SmallComputer System Interface (SCSI) protocol. The proxy 105 uses thesecontrol operations to determine when to invalidate or clear data intiering media 110. A first SCSI bus rescan operation enumerates alldevices on the SCSI bus. The rescan operation references each device onthe SCSI bus and is used for adding devices to the SCSI bus or toidentity a removed device. The rescan operation is typically performedafter a snapshot operation when the media devices 120 are remounted inblock 310 in FIG. 4.

TABLE 1 CONTROL OPERATIONS NUMBER OF MEDIA TYPE PURPOSE DEVICESREFERENCED SCSI BUS ENUMERATE ALL ALL DEVICES ON SCSI RESCAN DEVICES ONSCSI BUS BUS SCSI DEVICE OBTAIN DEVICE ONLY SPECIFIED INQUIRY PARAMETERSFOR DEVICE SPECIFIC SCSI DEVICE

A second SCSI device inquiry message obtains parameters for specifiedSCSI target devices that have already been scanned and applies to theSCSI devices specifically referenced in the device inquiry message. Forexample, the SCSI bus rescan indicates a particular number of mediadevices 120 in the storage media 114 and the SCSI device inquiryidentifies the size and other parameters of the individual media devices120.

The SCSI bus rescan is typically associated with a completereconfiguration of a SCSI device. However, SCSI device inquiry canhappen at any time and is not necessarily associated the reconfigurationof a SCSI device. For example, an initiator may issue a SCSI deviceinquiry to check the status of a target device. The exact cases duringwhich rescan and inquiry operations occur depend on the operating systemof the client and the exact configuration of the operating system andapplications software.

FIG. 5 shows one embodiment of the storage access system 100 thatmonitors control operations 103 sent between the client devices 106 andstorage media 114, in addition to the read and write memory accessoperations 102 described above in FIG. 1. In one example, the controloperations 103 include SCSI commands for the SCSI protocol used over aSCSI fiber channel network 112. However, the control operations 103could be any operations used in any protocol that can be associated withpotentially non-concurrent data in tiering media 110.

The storage access system 100 includes registers, buffers, or memorythat stores configuration data 107. The configuration data 107 is usedby the proxy 105 to determine when to clear or invalidate data intiering media 100. The configuration information 107 can be entered by asystem administrator based on the type of control operations 103performed in the system in FIG. 5. The configuration information 107 canalso be dynamically changed, for example using a script or ApplicationProgrammers Interface (API) according to the particular controloperations 103 currently being performed on the SCSI bus 112 and/orbased on the frequency of the control operations 103.

Referring to FIG. 6, in one embodiment, the proxy 105 in block 702detects control operations 103 sent from the client device 106 to thestorage media 114. Again in one example, the control operations 103 areSCSI messages. The proxy 105 in block 704 checks to see if the controloperation 103 is a SCSI bus rescan operation. For example, the proxy 105looks for a designator in SCSI control messages that indicate a busrescan message. If the message is not a bus rescan, the proxy 105continues to monitor the control operations in block 702.

If the control operation 103 is a bus rescan in block 704, the proxy 105in block 706 invalidates all of the data in tiering media 706. The proxy105 assumes that the bus rescan operation 103 followed some offlineoperation that possibly changed the data in storage media 114. Forexample, the bus rescan could have followed the snapshot operationdescribed in FIG. 2. Accordingly, the proxy 105 invalidates all of thedata in tiering media 110 to prevent out of date data from beingsupplied to the client application 116.

In some computer systems, client devices 106 may assume that the mediadevices 120 maintain the same configuration after a snapshot operation.Accordingly, the client devices 106 may not issue bus rescans aftersnapshot operations or after other offline operations. If there is noSCSI bus rescan, the proxy 105 will not clear the data in tiering media110 and could supply out of date data to the client device 106.

The proxy device 105 could be programmed to clear the tiering media 110after some other SCSI operation affiliated with an offline operationthat changes data in storage media 114. For example, the proxy device105 could be programmed to clear the tiering media 110 responsive to theSCSI device inquiry message described above in Table 1.0. Referringbriefly back to FIG. 4, the client device 106 issues the SCSI deviceinquiry after a snapshot operation and before the media devices 120 areremounted in operation 310.

However, the client devices 106 may frequently issue SCSI deviceinquires to the media devices 120 to obtain device status information.Frequently clearing the tiering media 110 after each SCSI device querywould substantially slow down the storage access system 100. If the datain tiering media 110 is frequently invalidated, the storage accesssystem 100 could not provide as many hits from the faster memory devicescontained in tiering media 110. The storage access system 100 could evenslow memory access times below the typical speeds provided by storagemedia 114.

FIG. 7 shows how the storage access system 100 ensures correct data isprovided to the client devices 106 and also prevents invalidation of thedata in tiering media 110 from significantly slowing down memory accesstimes. The proxy 105 in block 802 monitors the SCSI control operations103 exchanged between the client device 106 and storage media 114. Theproxy 105 in block 804 checks to see if the control operation 107 is aSCSI bus rescan. If the control operation 103 is a bus rescan in block804, the proxy 105 in block 806 invalidates all of the data in tieringmedia 706. This prevents the storage access system 100 from providingout of date data when the client application 116 makes subsequent memoryaccess requests 102 (FIG. 1) to storage media 114.

If the control operation 103 is not a SCSI bus rescan, proxy 105 inblock 808 checks to see if the control operation 103 is a SCSI deviceinquiry. If the control operation 103 is not a SCSI device inquiry, theproxy 105 goes back to monitoring the control operations 103 in block802. If the control operation 103 is a SCSI device inquiry, the proxy105 in block 810 checks the configuration data 107 in block 810.Alternatively, the proxy 105 could have also checked the configurationdata 107 earlier during initial device configuration.

As explained above, different computer systems may perform SCSI busrescans and SCSI device inquires in different situations. For example,some computing systems may not perform snapshot operations. Othercomputer systems may decide to issue the SCSI device inquires inconjunction with the mounting of media devices after snapshotoperations.

An administrator or client device 106 programs the configuration data107 in a register or memory device. The configuration data 107 eitherenables or disables the proxy 105 to invalidate data in tiering media110. The configuration data 107 may remain static during subsequentsystem operations or the administrator or client device 106 maydynamically set or change the configuration data 107 when a snapshotoperation is performed.

The proxy device reads the configuration data 107 in block 810 todetermine if SCSI device inquiries are associated with an operation,such as a snapshot operation, that requires invalidation of at leastsome data in tiering media 110. For example, the configuration data 107may be a bit or flag that is set to notify the proxy 105 to clear datain the tiering media 110 whenever a SCSI device inquiry is detected. Theconfiguration data 107 can be set via an administration script based ona time of day, initiation of a snapshot operation, or based on any otherevent that can change coherency between data in storage media 114 anddata in tiering media 110.

If the configuration data 107 is not set in block 810, the proxy 105moves back to block 802 and waits for the next control operation.Otherwise, the proxy 105 in block 812 invalidates the data in tieringmedia 110 associated with the particular media device 120 identified inthe SCSI device inquiry.

For example, data in tiering media 110 is mapped to a particular mediadevice 120 and to a particular address or block address in the mediadevice 120. The proxy 105 searches for any data in tiering media 110that maps to the media device 120 identified in the SCSI device inquiry.The proxy 105 then invalidates the identified data or blocks of data inoperation 812. In another example, the device referenced in the SCSIdevice inquiry may represent multiple disks or a stripe of data acrossmultiple disks in a device volume. The proxy 105 in operation 812 onlyinvalidates the data in tiering media 110 associated with thoseparticular disks or device volume.

Thus, outdated data is invalidated in the tiering media 110 even whenthe client device 106 fails to issue SCSI bus rescans after snapshotoperations. Invalidation based on SCSI devices inquiries isprogrammable. Therefore, the proxy 105 will also not unnecessarilyinvalidate data in the tiering media 110 for SCSI device inquiries notassociated with snapshot operations or for other operations that do notrequire invalidation of the data in tiering media 110.

Several preferred examples have been described above with reference tothe accompanying drawings. Various other examples of the application arealso possible and practical. The system may be exemplified in manydifferent forms and should not be construed as being limited to theexamples set forth above.

The figures listed above illustrate preferred examples of theapplication and the operation of such examples. In the figures, the sizeof the boxes is not intended to represent the size of the variousphysical components. Where the same element appears in multiple figures,the same reference numeral is used to denote the element in all of thefigures where it appears.

Only those parts of the various units are shown and described which arenecessary to convey an understanding of the examples to those skilled inthe art. Those parts and elements not shown may be conventional andknown in the art.

The system described above can use dedicated processor systems, microcontrollers, programmable logic devices, or microprocessors that performsome or all of the operations. Some of the operations described abovemay be implemented in software and other operations may be implementedin hardware.

For the sake of convenience, the operations are described as variousinterconnected functional blocks or distinct software modules. This isnot necessary, however, and there may be cases where these functionalblocks or modules are equivalently aggregated into a single logicdevice, program or operation with unclear boundaries. In any event, thefunctional blocks and software modules or features of the flexibleinterface can be implemented by themselves, or in combination with otheroperations in either hardware or software.

What is claimed is:
 1. A method, comprising: storing, by a computingdevice, at least some data from a storage media in a tiering media;monitoring, by the computing device, memory access requests from clientsto the storage media; supplying, by the computing device, the data inthe tiering media corresponding to the memory access requests to theclients; monitoring, by the computing device, control operationsexchanged between the clients and the storage media; identifying, by thecomputing device, a device inquiry in the control operations; andselectively invalidating, by the computing device, at least some of thedata in the tiering media responsive to identifying the device inquiry.2. The method of claim 1, further comprising: identifying a media devicein the storage media identified in the device inquiry; and invalidatingonly the data in the tiering media mapping to the media device in thestorage media.
 3. The method of claim 1, further comprising: referencingconfiguration information; and invalidating the data in the tiring mediaaccording to the configuration information.
 4. The method of claim 3,wherein the configuration information programmably enables invalidationof data in the tiering media when the device inquiry is detected.
 5. Themethod of claim 1, further comprising: monitoring Small Computer SystemInterface (SCSI) communications between the clients and the storagemedia to identify a SCSI device inquiry; and selectively invalidating atleast some of the data in the tiering media when the SCSI device inquiryis detected.
 6. The method of claim 1, further comprising: monitoringfor a bus rescan message in the control operations; and invalidating allof the data in the tiering media when the bus rescan message isidentified.
 7. The method of claim 6, further comprising: monitoringSmall Computer System Interface (SCSI) communications between theclients and the storage media; and invalidating all of the data in thetiering media when a SCSI bus rescan message is detected in the SCSIcommunications.
 8. The method of claim 1, wherein the device inquiry isassociated with a prior snapshot operation.
 9. An apparatus, comprising:tiering media coupled between clients and storage media; a configurationdevice configured with a parameter; and a processor coupled between theclients and the storage media, the processor configured to monitormemory access operations by the clients, store at least some dataassociated with the memory access operations in both the storage mediaand the tiering media, and provide the data from the tiering media tothe clients according to the memory access operations; the processorfurther configured to monitor Small Computer System Interface (SCSI)communications between the clients and the storage media and invalidateat least some of the data in the tiering media when a particular SCSImessage in the SCSI communications is detected and the parameter in theconfiguration device associates the SCSI message with an invalidationcondition for the tiering media.
 10. The apparatus according to claim 9,wherein the processor is further configured not to invalidate data inthe tiering media when the parameter in the configuration device doesnot associate the SCSI message with an invalidation condition.
 11. Theapparatus according to claim 9, wherein the SCSI message is a deviceinquiry.
 12. The apparatus according to claim 9, wherein theconfiguration device is a buffer or memory in a proxy device.
 13. Theapparatus according to claim 12, wherein the processor is furtherconfigured to identify a disk device in the storage media referenced inthe SCSI communications and only invalidate data in the tiering mediathat maps to the disk device.
 14. The apparatus according to claim 11,wherein the processor is further configured to invalidate all of thedata in the tiering media whenever a SCSI bus rescan is detected in theSCSI communications regardless of what parameter is configured in theconfiguration device.
 15. The apparatus according to claim 9, furthercomprising a storage controller configured to operate independently fromthe processor and perform snapshot operations of the data in the storagemedia and wherein the snapshot operations are hidden from the processor,and wherein the processor is configured to invalidate data in thetiering media responsive to the snapshot operations by monitoring theSCSI communications between the clients and the storage media.
 16. Theapparatus according to claim 9, wherein the processor and the tieringmedia are both part of storage access system that operates as a proxyfor the memory access operations from the clients to the storage media.17. A storage access system, comprising: a proxy device configured tomonitor memory access requests from clients to a disk array andselectively copy data into a tiering media for servicing some of thememory access requests, wherein the proxy device is further configuredto monitor control operations between the clients and the disk array andselectively invalidate at least some data in the tiering media when thecontrol operations indicate snapshot operations were performed.
 18. Thestorage access system according to claim 17, wherein the proxy device isconfigured to invalidate at least some of the data in the tiering mediaresponsive to a Small Computer System Interface (SCSI) messageidentified in the control operations.
 19. The storage access systemaccording to claim 18, further comprising a programmable configurationdevice configured to enable the proxy device to invalidate some of thedata in the tiering media when a first SCSI device inquiry message isidentified in the control operations.
 20. The storage access systemaccording to claim 19, wherein the proxy device is configured to:invalidate all data in the tiering media when a SCSI bus rescan isidentified in the control operations; and invalidate only a portion ofthe data in the tiering media when the SCSI device inquiry is identifiedin the control operations, wherein the portion of the data invalidatedmaps to a disk device in the disk array identified in the SCSI deviceinquiry.